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Abstract — The value 1 problem is a decision problem for 
probabilistic automata over finite words: given a probabilistic 
automaton A, are there words accepted by A with probability 
arbitrarily close to 1? 

This problem was proved undecidable recently. We sharpen 
this result, showing that the undecidability holds even if the 
probabilistic automata have only one probabilistic transition. 

Our main contribution is to introduce a new class of proba- 
bilistic automata, called leaktight automata, for which the value 
1 problem is shown decidable (and PSPACE-complete). We 
construct an algorithm based on the computation of a monoid 
abstracting the behaviors of the automaton, and rely on algebraic 
techniques developed by Simon for the correctness proof. The 
class of leaktight automata is decidable in P SPACE, subsumes 
all subclasses of probabilistic automata whose value 1 problem 
is known to be decidable (in particular deterministic automata), 
and is closed under two natural composition operators. 

Introduction 

Probabilistic automata: Rabin invented a very simple yet 
powerful model of probabilistic machine called probabilistic 
automata, which, quoting Rabin, "are a generalization of finite 
deterministic automata" |fl9l . A probabilistic automaton has 
a finite set of states Q and reads input words over a finite 
alphabet A. The computation starts from the initial state i 
and consists in reading the input word sequentially; the state 
is updated according to transition probabilities determined 
by the current state and the input letter. The probability to 
accept a finite input word is the probability to terminate the 
computation in one of the final states F C Q. 

From a language-theoretic perspective, several algorithmic 
properties of probabilistic automata are known: while language 
emptiness is undecidable J2], JT3), ifTSl . language equivalence 
is decidable 0, EOJ, E3 as well as other properties 0,0. 

Rather than formal language theory, our initial motivation 
for this work comes from control and game theory: we aim 
at solving algorithmic questions about partially observable 
Markov decision processes and stochastic games. For this 
reason, we consider probabilistic automata as machines con- 
trolled by a blind controller, who is in charge of choosing the 
sequence of input letters in order to maximize the acceptance 
probability. While in a fully observable Markov decision 
process the controller can observe the current state of the 
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process to choose adequately the next input letter, a blind 
controller does not observe anything and its choice depends 
only on the number of letters already chosen. In other words, 
the strategy of a blind controller is an input word of the 
automaton. 

The value of a probabilistic automaton: With this game- 
theoretic interpretation in mind, we define the value of a proba- 
bilistic automaton as the supremum of acceptance probabilities 
over all input words, and we would like to compute this value. 
Unfortunately, as a consequence of an undecidability result due 
to Paz, the value of an automaton is not computable in general. 
However, the following decision problem was conjectured by 
Bertoni to be decidable 0: 

Value 1 problem: Given a probabilistic automaton, does 
the automaton have value 1 ? In other words are there input 
words whose acceptance probability is arbitrarily close to 1 ? 

Actually, Bertoni formulated the value 1 problem in a 
different yet equivalent way: "Is the cut-point 1 isolated or 
not?". There is actually a close relation between the value 1 
problem and the notion of isolated cut-point introduced by 
Rabin in the very first paper about probabilistic automata. A 
real number < A < 1 is an isolated cut-point if there exists 
a bound e > such that the acceptance probability of any 
word is either greater than A + e or smaller than A — e. A 
theorem of Rabin states that if the cut-point A is isolated, then 
the language L\ = {w | Pa( w ) > A} is regular [19|. The 
value 1 problem can be reformulated in term of isolated cut- 
point: an automaton has value 1 if and only if 1 is not an 
isolated cut-point. Bertoni proved that for A strictly between 
and 1, the isolation of A is undecidable in general, and left 
the special case A 6 {0, 1} open. 

Recently, the second and third authors of the present paper 
proved that the value 1 problem is undecidable lfl3l as 
well. However, probabilistic automata, and more generally 
partially observable Markov decision processes and stochastic 
games, are a widely used model of probabilistic machines 
used in many fields like software verification 0, 0, image 
processing iflOl . computational biology ifTTI and speech pro- 
cessing irTTl . As a consequence, it is crucial to understand 
which decision problems are algorithmically tractable for 
probabilistic automata. 



Our result: As a first step, we sharpen the undecidability 
result: we prove that the value 1 problem is undecidable 
even for probabilistic automata with only one probabilistic 
transition. This result motivated the introduction of a new class 
of probabilistic automata, called leaktight automata, for which 
the value 1 problem is decidable. This subclass subsumes 
all known subclasses of probabilistic automata sharing this 
decidability property and is closed under parallel composition 
and synchronized product. Our algorithm to decide the value 1 
problem computes in polynomial space a finite monoid whose 
elements are directed graphs and checks whether it contains a 
certain type of elements that are value 1 witnesses. 

Related works: The value 1 problem was proved decid- 
able for a subclass of probabilistic automata called jj-acyclic 
automata [13]. Since the class of jj-acyclic automata is strictly 
contained in the class of leaktight automata, the result of the 
present paper extends the decidability result of IfPJl . Chadha et 
al. (3:| recently introduced the class of hierarchical probabilistic 
automata, which is also strictly contained in the class of 
leaktight automata. As a consequence of our result, the value 
1 problem is decidable for hierarchical probabilistic automata. 
Our proof techniques totally depart from the ones used in fl3j , 
[ 1 3 1 . Instead, we make use of algebraic techniques and in 
particular Simon's factorization forest theorem, which was 
successfully used to prove the decidability of the boundedness 
problem for distance automata ||221 . 

Outline: We give the basic definitions in Section Q] As 
a first step we present our algorithm to decide the value 
1 problem of probabilistic leaktight automata in Section [II] 
which is followed by the decidability of the leaktight property 
in Section Hill Next, in Section II VI we present and prove the 
technical core of the paper, called the lower bound lemma. 
Finally, Section [V] investigates properties and provides exam- 
ples of leaktight automata. The proofs can be found in the 
appendix. 

I. Definitions 

A. Probabilistic automata 

Let Q be a finite set of states. A probability distribution 
over Q is a row vector i5 of size | Q | whose coefficients are real 
numbers from the interval [0, 1] and such that J2 q ^Q ^(l) = ^■ 
A probabilistic transition matrix M is a square matrix in 
[0, 1] < 3 X( 3 such that every row of M is a probability distri- 
bution over Q. 

Definition 1 (Probabilistic automata). A probabilistic automa- 
ton A is a tuple (Q, A,(M a ) a< zA,h F), where Q is a finite 
set of states, A is the finite input alphabet, (M a ) ae A ore the 
probabilistic transition matrices, i € Q is the initial state and 
F C Q is the set of accepting states. 

For each letter a G A, M a (s,t) is the probability to go 
from state s to state t when reading letter a. Given an input 
word w G A*, we denote by w(s,t) the probability to go 
from state s to state t when reading the word w. Formally, 
if w = aia 2 ■■■a n then w(s, t) = (M ai ■ M a2 ■ ■ ■ M a „)(s, t). 
Note that < w(s, t) < 1, for all words w and states s and t. 



Furthermore, the definition of a probabilistic transition matrix 
implies that: YlteQ w ( s > *) = 1 f° r a ^ states s - 

Definition 2 (Value and acceptance probability). The accep- 
tance probability of a word w G A* by A is ¥^(w) = 
S/eF /)■ The value of A denoted val(A), is the supre- 
mum of the acceptance probabilities over all possible input 
words: 

val(A) = sup F A (w) . (1) 

B. The value 1 problem for probabilistic automata 
We are interested in the following decision problem: 

Problem (Value 1 Problem). Given a probabilistic automaton 
A decide whether val(A) = 1. 

The value 1 problem can be reformulated using the notion 
of isolated cut-point introduced by Rabin in his seminal 
paper lfl9l : an automaton has value 1 if and only if the cut- 
point 1 is not isolated. 

Whereas the formulation of the value 1 problem only relies 
qualitatively on the asymptotic behavior of probabilities (the 
probability to be in non-final states should be arbitrarily small) 
the answer to the value 1 problem depends quantitatively on 
the transition probabilities. 
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Fig. 1. This automaton has value 1 if and only if x > ^. 

For instance, the automaton depicted on Fig. [TJ has value 
1 if and only if x > \ and has value less or equal than I 
otherwise, see also H], |[l3l for similar results. Note that in this 
example, the value is a discontinuous function of the transition 
probabilities. The input alphabet is A = {a, b}, the initial state 
is the central state and the unique final state is T. In order to 
maximize the probability to reach T, playing two 6's in a row 
is certainly not a good option because from state this ensures 
to reach the non-accepting absorbing state _L with probability 
at least |. A smarter strategy consists in playing one b, then 
long sequences of a's followed by one letter b. If x < |, 
there is still no hope to have a word accepted with probability 
strictly greater than i: starting from 0, and after a b and a 
sequence of a's, the probability to be in R is greater or equal 
than the probability to be in L, thus playing ba n b from state 
the probability to reach the sink _L is greater or equal than 
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the probability to reach the final state T. However, if x > \ 
then a simple calculation shows that the probability to accept 
(ba n ) n tends to 1 as n goes to infinity. 

C. Undecidability in a very restricted case 

As a first step we refine the undecidability result: we show 
that the value 1 problem is undecidable even when restricted 
to probabilistic automata having exactly one probabilistic 
transition. For such automata, there exists exactly one state 
s and one letter a such that < M a {s,t) < 1 for all t 
and the remaining transitions are deterministic: for all triple 
(s',a',t) e S x A x S such that (s',a') ^ (s,a) then 
M a (s',t) e {0,1}. 

The general idea is to simulate any probabilistic automaton 
A with a probabilistic automaton B which has only one 
probabilistic transition and such that val(.4) = 1 if and only 
if val(B) = 1. 

As a first attempt, we define the automaton B with a larger 
alphabet: whenever A reads a letter a, then B reads a sequence 
of actions a corresponding to a, allowing a state-by-state 
simulation of A. The unique probabilistic transition of B is 
used to generate random bits for the simulation. However, the 
automaton B cannot check that the sequences of actions are 
well-formed and allow for a faithful simulation. Hence we 
modify the construction, such that to simulate the automaton 
A on the input word w, the automaton B now reads (w) n for 
arbitrarily large n. Each time B reads a word w, it simulates 
A on w with a small yet positive probability and "delays" the 
rest of the simulation, also with positive probability. This delay 
process allows to run on parallel a deterministic automaton 
which checks that the sequences of actions are well-formed, 
ensuring a faithful simulation. The complete details can be 
found in the appendix (see also [12]). 

This undecidability result illustrates that even very restricted 
classes of probabilistic automata may have an undecidable 
value 1 problem. In the next section, we introduce a non-trivial 
yet decidable subclass of probabilistic automata, defined by the 
leaktight property. 

D. Informal description of the leaktight property 

One of the phenomena that makes tracking vanishing prob- 
abilities difficult are leaks. A leak occurs in an automaton 
when a sequence of words turns a set of states C C Q into a 
recurrence class C on the long run but on the short run, some 
of the probability of the recurrence class is "leaking" outside 
the class. 

Such leaks occur in the automaton of Fig. [T]with the input 
sequence (a n b) n( zm. As n grows large, the probability to reach 
T and _L while reading the input word a n b vanishes: there 
are leaks from L to T and symmetrically from R to _L. As 
a consequence, the real asymptotic behavior is complex and 
depends on the compared speeds of these leaks. 

An automaton without leak is called a leaktight automaton. 
In the next section we prove that the value 1 problem is 
decidable when restricted to the subclass of leaktight automata. 



The definition of a leaktight automaton relies on two key 
notions, idempotent words and word-recurrent states. 

A finite word u is idempotent if reading once or twice the 
word u does not change qualitatively the transition probabili- 
ties: 

Definition 3 (Idempotent words). A finite word u £ A* is 
idempotent if for every states s,t £ Q, 

u(s,t) > (u-u)(s,t)>0 . 

Idempotent words are everywhere: every word, if iterated a 
large number of times, becomes idempotent. 

Lemma 1. For every word u 6 A*, the word is 
idempotent. 

A finite word u induces naturally a finite homogeneous 
Markov chain, which splits the set of states into two classes: 
recurrent states and transient states. Intuitively, a state is 
transient if there is some non-zero probability to leave it 
forever, and recurrent otherwise; equivalently from a recurrent 
state the probability to visit it again in the future is one. 

Definition 4 (Recurrent states). Let u G A* be a finite word. A 
state s is u-recurrent if it is recurrent in the finite Markov chain 
M. u with states Q and transitions probabilities (u(s, t)) s jeQ- 

Formally, s in recurrent in AA U if for all t in Q, if there 
is a non-zero probability to reach t from s, then there is a 
non-zero probability to reach s from t. 

In the case of idempotent words, recurrence of a state can 
be easily characterized: 

Lemma 2. Let s be a state and u be an idempotent word. 
Then s is u-recurrent if for every state t, 

u(s,t) > u(t,s) > . 

The proof of this lemma follows from the observation that 
since u is idempotent, there is a non-zero probability to reach 
t from s if and only if u(s, t) > 0. 

The formal definition of a leak is as follows: 

Definition 5 (Leaks and leaktight automata). A leak from a 
state r £ Q to a state q £ Q is a sequence (u n ) n< =fj of 
idempotent words such that: 

1) for every s,t 6 Q, the sequence (w n (s, i))„ e N con- 
verges to some value u{s, t). We denote by A4 U the 
Markov chain with states Q and transition probabilities 
(u(s,t)) S! teQ, 

2) the state r is recurrent in M. u , 

3) for all n in N, u n {r, q) > 0, 

4) and r is not reachable from q in Ai u . 

A probabilistic automaton is leaktight if it has no leak. 

The automaton depicted in Fig. Q] is not leaktight when < 
x < 1 because the sequence (u„) ji(E n = (a n b) ne -^ is a leak 
from L to T, and from R to _L The limit Markov chain M u 
sends state to states L and R with probability i each, and 
all other states are absorbing (i.e loop with probability 1). In 
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particular, state L is recurrent in A4 U , for every n, u n (L, T) > 
but there is no transition from T to L in A4 U . 

Several examples of leaktight automata are given in Sec- 
tion El 

II. The value 1 problem is decidable for leaktight 

AUTOMATA 

In this section we establish our main result: 

Theorem 1. The value 1 problem is decidable for leaktight 
automata. 

A. The Markov monoid algorithm 

Our decision algorithm for the value 1 problem computes 
iteratively a set Q of directed graphs called limit- words. Each 
limit-word is meant to represent the asymptotic effect of a 
sequence of input words, and some particular limit-words can 
witness that the automaton has value 1. 



Algorithm 1 The Markov monoid algorithm. 

Input: A probabilistic automaton A. 

Output: Decide whether A has value 1 or not. 

1 Q <r- {a | a G A}U {1}. 

2 repeat 

3 if there is u, v G G such that u • v ^ Q then 

4 add u • v to Q 

5 if there is u G Q such that u = u • u and u" ^ Q 
then 

6 add u" to Q 

1 until there is nothing to add 

8 if there is a value 1 witness in Q then 

9 return true 

10 else 

1 1 return false 



In the rest of the section, we explain the algorithm in details. 

Definition 6 (Limit-word). A limit-word is a map u : Q 2 — > 

{0, 1} such that Vs G Q, 3t G Q, u(s, t) = 1. 

The condition expresses that our automata are complete: 
whatever the input word, from any state s there exists some 
state t which is reached with positive probability. A limit-word 
u can be seen as a directed graph with no dead-end, whose 
vertices are the states of the automaton A, where there is an 
edge from s to t if u(s, t) = 1. 

Initially, Q only contains those limit-words a that are 
induced by input letters a G A, where the limit-word a is 
defined by: 

Vs,t G Q, (a(s,i) = 1 a(s,t)>0) . 

plus the identity limit-word 1 defined by (l(s,i) = 1) 

(s = t), which represents the constant sequence of the empty 

word. 

The algorithm repeatedly adds new limit-words to Q. There 
are two ways for that: concatenating two limit-words in Q or 
iterating an idempotent limit-word in Q. 



Concatenation of two limit-words: The concatenation of 
two limit-words u and v is the limit-word u ■ v such that: 

(u • v)(s, t) = 1 3q G Q, u(s, q) = 1 and v(q, t) = 1 . 

In other words, concatenation coincides with the multipli- 
cation of matrices with coefficients in the boolean semiring 
({0, 1}, V, A). The concatenation of two limit-words intu- 
itively corresponds to the concatenation of two sequences 
(u n )n£N and (w,i)„gN of input words into the sequence 
(u n ■ v,i)neN- Note that the identity limit-word 1 is neutral 
for the concatenation. 

Iteration of an idempotent limit-word: Intuitively, if a 
limit-word u represents a sequence (u n ) n£ n then its iteration 
u" represents the sequence [Un ) for an arbitrarily large 

V / n£N 

increasing function / : N — > N. 

The iteration u" of a limit-word u is only defined when u 
is idempotent i.e when u ■ u = u. It relies on the notion of 
u-recurrent state. 

Definition 7 (u-recurrence). Let u be an idempotent limit- 
word. A state s is u-recurrent if for every state t, 

u(s,t) = 1 ==> u(t, s) = 1 . 

The iterated limit-word removes from u any edge that 
does not lead to a recurrent state: 

u"(s,i) = 1 <^=^ u(s,i) = 1 and t is u-recurrent . 

B. The Markov monoid and value 1 witnesses 

The set Q of limit-words computed by the Markov monoid 
algorithm is called the Markov monoid. 

Definition 8 (Markov monoid). The Markov monoid is the 
smallest set of limit-words containing the set {a | a G A} of 
limit-words induced by letters, the identity limit-word 1, and 
closed under concatenation and iteration. 

Two key properties, consistency and completeness, ensure 
that the limit-words of the Markov monoid reflect exactly 
every possible asymptotic effect of a sequence of input words. 

Consistency ensures that every limit-word in Q abstracts the 
asymptotic effect of an input sequence. 

Definition 9 (Consistency). A set of limit-words Q C {0, l}^ 2 
is consistent with a probabilistic automaton A if for each limit- 
word u G Q, there exists a sequence of input words (u„) ng N 
such that for every states s,t G Q the sequence (u n (s, t)) n eN 
converges and 

u(s,t) = l ]jmu n (s,t) > . (2) 

n 

Conversely, completeness ensures that every input sequence 
reifies one of the limit-words. 

Definition 10 (Completeness). A set of limit-words Q C 
{0,1}*^ is complete for a probabilistic automaton A if for 
each sequence of input words (u n ) n £N, there exists u G Q 
such that for every states s,t G Q: 

limsupu„(s, t) = => u(s,<)=0 . (3) 

n 
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A limit- word may witness that the automaton has value 1. 

Definition 11 (Value 1 witnesses). Let A be a probabilistic 
automaton. A value 1 witness is a limit-word u such that for 
every state s G Q, 

u(i, s) = 1 s € F . (4) 

Thanks to value 1 witnesses, the answer to the value 1 
problem can be read in a consistent and complete set of limit- 
words: 

Lemma 3 (A criterion for value 1). Let A be a probabilistic 
automaton and Q C {0, 1}^ be a set of limit-words. Suppose 
that Q is consistent with A and complete for A Then A has 
value 1 if and only if Q contains a value 1 witness. 

In order to illustrate the interplay between limit-words of 
the Markov monoid and sequences of input words, we give a 
detailed proof of Lemma [3] 

Proof: Assume first that A has value 1. By 
definition, there exists a sequence {u n ) ne fi of input 
words such that P^.(tt n ) — >n L As a consequence, 
12feF u n{hf) — ^A( u n) — >n 1- Since for all n G N, we 
have J2 q eQ u n(h q) = 1* then f ° r a U s ' £ F, u n (i, s') — > n 0. 
Since Q is complete, there exists a limit-word u such that (01 
holds. Then u is a value 1 witness: for every s G Q such 
that u(i,s) = 1, equation (f3]i implies limsup n u n {i, s) > 0, 
hence s G F. 

Conversely, assume now that Q contains a value 1 witness 
u. Since Q is consistent, there exists a sequence (u„)„eN such 
that © holds. It follows from © and ©, that for all s g F, we 
have u n (i, s) — > n 0. Thus F A (u n ) = J2feF u n{h /) — >n 1 
and A has value 1, ■ 

The following theorem proves that the Markov monoid of a 
leaktight automaton is consistent and complete, thus according 
to Lemma [3] it can be used to decide the value 1 problem. 

Theorem 2. The Markov tnonoid associated with an automa- 
ton A is consistent. Moreover if A is leaktight then the Markov 
monoid is complete. 

The proof of the second part of this theorem relies on a sub- 
tle algebraic argument based on the existence of factorization 
forests of bounded height [21 1. The same kind of argument was 
used by Simon to prove the decidability of the boundedness 
problem for distance automata ||221 . 

We postpone the proof of completeness to the next section, 
where a slightly more general result is established; for now 
we show that the Markov monoid is consistent. 

Lemma 4 (Consistency). Let Q C {0, 1}® be a set of limit- 
words. Suppose that Q is consistent. Then for every u,v£5 
the set Q U {u • v} is consistent. If moreover u is idempotent 
then Q U {u"} is consistent as well. 

The proof uses the notion of reification. 

Definition 12. A sequence (itn)neN of input words reifies a 
limit-word u if for every states s, t the sequence (u n (s, f)) n eN 



converges and 

u(s,f) = l ]imu n (s,t) > . (5) 

n 

In particular, a set of limit-words Q is consistent for A if each 
limit-word in Q is reified by some sequence of input words. 

Proof: Let u, v G Q. We build a sequence (w n ) n ^ which 
reifies u • v. By induction hypothesis on u and v, there exists 
(u n )n and (v„) n which reify u and v respectively. Let w n = 
u n ■ v n . Then (w n )nefi reifies u • v, because 

w n (s, r) = ^ U «( S ' *) " w n(^ r ) 

and by definition of the concatenation of two limit-words. 

Suppose now that u is idempotent, we build a sequence 
(z„)„£s which reifies u". By induction hypothesis, there 
exists a sequence (M n )neN which reifies u. For every states 
s, t we denote by u(s,t) the value lim n u n (s,t). Since u is 
idempotent, the Markov chain M. u with state space Q and 
transition probabilities (u(s, t)) Syt £Q is 1-periodic thus aperi- 
odic. According to standard results about finite Markov chains, 
the sequence of matrices (u fc )/ ce N has a limit z G [0, V\Q X Q 
such that transient states of M. u have no incoming edges in 
z. This implies: 

Vs, t G Q, (z(s,t) > t is z-recurrent) . (6) 

Since (u n )neN converges to u and by continuity of the matrix 
product, for every k G N the sequence of matrices (uJj)„ e N 
converges to u k . It follows that there exists (f>(k) G N such 
that ||u fc — w^-jHoo < i. As a consequence the sequence of 
matrices (z n ) neN = («$(7i))«eN converges to z. 
Now we prove that (z n ) n eN reifies u" because, 

ir(s,f) = 1 t is u-recurrent and u(s,t) = 1 

<^=^ t is u-recurrent and u(s, t) > 

<^=> t is z-recurrent and z(s, t) > 

^ z(s,t) > 

<^=> lim z n (s,t) > , 

n 

where the first equivalence is by definition of the iteration, the 
second holds because (u n ) ne ti reifies u, the third because the 
iterated Markov chain induced by z = lirnfc u k has the same 
recurrent states than the Markov chain A4 U , the fourth holds 
by (O, and the fifth by definition of z. ■ 

C. Correctness of the Markov monoid algorithm 

Proposition 1. The Markov monoid algorithm solves the value 
1 problem for leaktight automata. 

Proof: Termination of the Markov monoid algorithm is 
straightforward because each iteration adds a new element in 
Q and there are at most 2^1 elements in Q. 

The correctness is a corollary of Theorem since the 
Markov monoid is consistent and complete then according to 
Lemma [3] A has value 1 if and only if Q contains a value 1 
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witness, if and only if the Markov monoid algorithm outputs 
"true". ■ 
In case the Markov monoid algorithm outputs "true", then 
for sure the input automaton has value 1, This positive result 
holds for every automaton, leaktight or not. 

Proposition 2. // the Markov monoid algorithm outputs 
"true ", the input probabilistic automaton has value 1. 

Proof: According to Theorem [2] the Markov monoid is 
consistent. If it contains a value 1 witness, then according to 
the second part of the proof of Lemma [3] A has value 1. ■ 
In case the Markov monoid algorithm outputs "false" and 
the automaton is leaktight then the value of the automaton can 
be bounded from above: 

Theorem 3. Let A be a probabilistic automaton whose 
minimal non-zero transition probability is denoted p mm . If the 
Markov monoid algorithm outputs "false " and if moreover A 
is leaktight, then val(A) < 1 — p^ n , with J — 2 2 l^l 2 . 

The proof of this theorem is postponed to the next section, 
because it relies on the notion of extended Markov monoid, 
it is actually a direct corollary of the lower bound lemma 
presented in Section ITVl 

In case the Markov monoid algorithm outputs "false", one 
surely wishes to know whether the input automaton is leaktight 
or not. Fortunately, the leaktight property is decidable, this is 
the subject of the next section. 

D. Complexity of the Markov monoid algorithm 

Proposition 3. The value 1 problem for leaktight automata is 
PSPACE-complete. 

The Markov monoid algorithm terminates in less than 2^^ 2 
iterations, since each iteration adds a new limit-word in the 
monoid and there are less than 2^1 different limit-words. 

This EXPTIME upper bound can be actually improved to 
PSPACE. For that we use the same arguments that Kirsten 
used to prove that limitedness of desert automata can be 
decided in PSPACE fl5l . 

A way to improve the complexity from EXPTIME 
to PSPACE is to avoid the explicit computation of 
the Markov monoid and to look for value 1 witnesses 
in a non-deterministic way. The algorithm guesses non- 
deterministic ally the value 1 witness u and its decomposition 
by the product and iteration operations. The algorithm com- 
putes a (J-expression, i.e a finite tree with concatenation nodes 
of arbitrary degree on even levels and iteration nodes of degree 
1 on odd levels and labelled consistently by limit-words. The 
depth of this tree is at most twice the ((-height (the number 
of nested applications of the iteration operation) plus 1. The 
root of the (J-expression is labelled by u and the expression is 
computed non-deterministically from the root in a depth-first 
way. 

For desert automata, the key observation made by Kirsten 
is that the ((-height is at most \Q\. The adaptation of Kirsten's 
proof to probabilistic automata relies on the two following 
lemmata: 



Lemma 5. Let u and v be two idempotent limit-words. 
Assume u <j v, then there are less recurrence classes in 
u than in v. 

Lemma 6. Let u be an idempotent limit-word. The set of 
recurrence classes of u is included in the set of recurrence 
classes of u". Moreover if u ^ this inclusion is strict. 

Since the number of recurrence classes in a limit-word is 
bounded by \Q\, and if we require the iteration operation to 
be applied only to unstable idempotent, the ((-height of a fl- 
expression is bounded by \Q\ thus the depth of the expression 
is bounded by 2\Q\ + 1. 

Consequently, the value 1 problem can be decided in 
PSPACE: to guess the value 1 witness, the non-deterministic 
algorithm needs to store at most 2\Q\ + 1 limit-words which 
can be done in space 0(\Q\ 2 ). Savitch's theorem implies that 
the deterministic complexity is PSPACE as well. 

This PSPACE-upperbound on the complexity is tight. The 
value 1 problem is known to be PSPACE-complete when 
restricted to jj-acyclic automata ifTJl . The same reduction to the 
PSPACE-complete problem of intersection of deterministic 
automata can be used to prove completeness of the value 1 
problem for leaktight automata, relying on the facts that de- 
terministic automata are leaktight (Proposition @J and the class 
of leaktight automata is closed under parallel composition 
(Proposition |5j. The completeness result is also a corollary 
of Proposition |4] since jj-acyclic automata are a subclass of 
leaktight automata, the decision problem is a fortiori complete 
for leaktight automata. 

III. Deciding whether an automaton is leaktight 

At first sight, the decidability of the leaktight property is 
not obvious: to check the existence of a leak one would need 
to scan the uncountable set of all possible sequences of input 
words. Still: 

Theorem 4. The leaktight property is decidable in polynomial 
space. 



Algorithm 2 The leak-finder algorithm. 
Input: A probabilistic automaton A. 
Output: Decide whether A is leaktight or not. 

1 0+<-{(a,a) |aeA}U{(l,l)}. 

2 repeat 

3 if there is (u, u+), (v, v + ) e 5+ such that (u-u, v + • 
v+) g G+ then 

4 add (u • v, u + • v+) to Q+ 

5 if there is (u, u + ) 6 Q + such that u = u • u and 
u + = u + • u + and (u", u + ) ^ Q + then 

6 add (uK u+) to Q + 
1 until there is nothing to add 

8 if there is a leak witness in Q + then 

9 return false 

10 else 

1 1 return true 
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The leak-finder algorithm deciding the leaktight property is 
very similar to the Markov monoid algorithm, except for two 
differences. First, the algorithm keeps track of those edges 
that are deleted by successive iteration operations. For that 
purpose, the algorithm stores together with each limit-word 
u another limit-word u + to keep track of strictly positive 
transition probabilities. Second, the algorithm looks for leak 
witnesses. 

Definition 13 (Extended limit-word). An extended limit-word 
is a pair of limit-words. The set of extended limit-words 
computed by the leak-finder algorithm is called the extended 
Markov monoid. 

The extended Markov monoid is indeed a monoid equipped 
with the component-wise concatenation operation: 

(u, u+) • (v, v+) = (u • v, u+ • v+) , 

It follows that an extended limit-word (u, u + ) is idempotent 
if both u and u + are idempotent. 

Definition 14 (Leak witness). An extended limit-word (u, u+) 
is a leak witness ;/ it is idempotent and there exists r,q G Q 
such that: 

1) r is u-recurrent, 

2) u + (r, g )=l, 

3) u(g,r) = 0. 

The correctness of the leak-finder algorithm is a conse- 
quence of: 

Theorem 5. An automaton A is leaktight if and only if its 
extended Markov monoid does not contain a leak witness. 

The proof can be found in the appendix. Although we chose 
to present Theorem [2] and Theorem [5] separately, their proofs 
are tightly linked. 

As a consequence, the leaktight property is qualitative: it 
does not depend on the exact value of transition probabilities 
but only on their positivity. 

IV. The lower bound lemma 

The lower bound lemma is the key to both our decidability 
result (via Theorem [3} and the characterization of leaktight 
automata (Theorem |5}. 

Lemma 7 (Lower bound lemma). Let A be a probabilistic 
automaton whose extended Markov monoid contains no leak 
witness. Let p m - m the smallest non-zero transition probability 
of A Then for every word u G A*, there exists a pair (u, u + ) 
in the extended Markov monoid such that, for all states s,t G 

Q, 

u+(s,f) = 1 u(s,t) > , (7) 

u(M) = 1 =>• <s,t) >p%£ , (8) 

where J = 2 2 I Q I 2 . 

To prove Lemma [7] we rely on the notion of Ramseyan 
factorization trees and decomposition trees introduced by 
Simon lETl, l22l. 



Definition 15. Let A be a finite alphabet, [M, •, 1) a monoid 
and (j> : A* — >• M a morphism. A Ramseyan factorization tree 
of a word u G A + for cf> is a finite unranked ordered tree, 
whose nodes are labelled by pairs (w,<f)(w)) where w is a 
word in A + and such that: 

(i) the root is labelled by (it, <fi{u)), 

(ii) every internal node with two children labelled by 
(iti, <p(ui)) and (it2, 0(1*2)) is labelled by (ui -112, </>(ui ■ 
u 2 )), 

(iii) leaves are labelled by pairs (a, 4>(a)) with a G A, 

(iv) if an internal node t has three or more children 
ti,...,t n labelled by (ui, cf)(ui)), . . . , (u n , (f>(u n )), then 
there exists e G M such that e is idempotent and 
e = 4>{u\) = 4>{u2) = ... = 4>{u n ). In this case t 
is labelled by {u\ ■ ■ ■ u n , e). 

Internal nodes with one or two children are concatenation 
nodes, the other internal nodes are iteration nodes. 

Not surprisingly, every word u G A + can be factorized 
in a Ramseyan factorization tree, using only concatenation 
nodes: any binary tree whose leaves are labelled from left to 
right by the letters of u and whose internal nodes are labelled 
consistently is a Ramseyan factorization tree. Notice that if 
u has length n then such a tree has height log 2 (n), with the 
convention that the height of a leaf is 0. As a consequence, 
with this naive factorization of u, the longer the word u, the 
deeper its factorization tree. 

The following powerful result of Simon states that every 
word can be factorized with a Ramseyan factorization tree 
whose depth is bounded independently of the length of the 
word: 

Theorem 6 ( lETl . J4), 0). Let Abe a probabilistic automaton 
whose extended Markov monoid contains no leak witness. 
Every word u G A + has a Ramseyan factorization tree of 
height at most 3 • |M|. 

In (22|, Simon used the tropical semiring (NU{oo}, min, +) 
to prove the decidability of the boundedness problem for 
distance automata. Similarly to the Markov monoid, the 
tropical semiring is equipped with an iteration operation jj. 
Following the proof scheme of Simon, we introduce the notion 
of decomposition tree relatively to a monoid M equipped with 
an iteration operation jj. 

Definition 16. Let A be a finite alphabet, (M, •, 1) a monoid 
equipped with a function jj that maps every idempotent e G M 
to another idempotent element e" G M and <fi : A* — > M a 
morphism. A decomposition tree of a word u G A + is a finite 
unranked ordered tree, whose nodes have labels in (A + , M) 
and such that: 

i) the root is labelled by (u, u), for some u G M, 

ii) every internal node with two children labelled by {u\, Ui) 
and (1*2, u 2 ) is labelled by (u± ■ U2, Ui • 112), 

iii) every leaf is labelled by (a, a) where a is a letter, 

iv) for every internal node with three or more children, there 
exists e G M such that e is idempotent and the node is 
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labelled by (iti . ..it„,e') and its children are labelled 
by (ui,e), (u„,e). 

Internal nodes with one or two children are concatenation 
nodes, the other internal nodes are iteration nodes. 

An iteration node labelled by (it, e) is discontinuous ife* ^ 
e. The span of a decomposition tree is the maximal length of 
a path that contains no discontinuous iteration node. 

Remark that decomposition and factorization trees are 
closely related: 

Lemma 8. A Ramseyan factorization tree is a decomposition 
tree if and only if it contains no discontinuous iteration nodes. 

Proof: The definitions [15] and [16] are similar except for 
condition iv). If there are no discontinuous nodes then e = e" 
in iv) of Definition [16] ■ 
The following theorem is adapted from [22, Lemma 10] and 
is a direct corollary of Theorem |6] 

Theorem 7. Let A be a finite alphabet, (M, •, 1) a monoid 
equipped with a function JJ that maps every idempotent e E M 
to another idempotent element e" G M and (f> : A* — > M 
a morphism. Every word it G A + has a decomposition tree 
whose span is less than 3 ■ |M|. 

To obtain the lower bound lemma, we need to bound the 
depth of a decomposition tree; now that the span is bounded 
thanks to Theorem [7] we need to bound the number of 
discontinuous iteration nodes. Simon and Leung noticed that 
this number is actually bounded by the number of j7-classes 
in the monoid. The notion of J'-class of a monoid M is a 
classical notion in semigroup theory, derived from one of the 
four Green's relations called the ./-preorder: a ./-class is an 
equivalence class for this preorder (for details about Green's 
relations, see [14|, [16|). The J'-preorder between elements 
of a monoid M is defined as follows: 

Vo, b G M, a < j b if a G MbM , 

where MbM denotes the set {ubv | it, v G M}. 

The number of discontinuous nodes along a path in a 
decomposition tree can be bounded using the following result, 
adapted from [22, Lemma 3]. 

Lemma 9. Let A be a finite alphabet, and M a monoid 
equipped with a function (j that maps every idempotent e G M 
to another idempotent element e" G M. Suppose moreover 
that for every idempotent e G M, 

e » . e = e* = e • e" . (9) 

Then for every idempotent element e G M, either e" = e or 

e" <j e. 

As a consequence, the number of discontinuous nodes along 
a path in a decomposition tree is at most J, where J is the 
number of J -classes of the monoid. 

Now we are ready to complete the proof of the lower bound 
lemma. 



Proof of Lemma [7] Let M be the extended Markov 
monoid Q + associated with A and equipped with the con- 
catenation operation: 

(u, u+) • (v, v+) = (u • v, u+ • v+) , 

and for idempotent pairs the iteration operation: 

(u,u+)» = (u»,u + ) . 

Let w G A + . (The case of the empty word is easily settled, 
considering the extended limit- word (1,1).) We apply Theo- 
rem [7] to the word w, the extended Markov monoid M = Q+ 
and the morphism tf> : A — >• M defined by <p(a) = (a, a). 
According to Theorem [7] w has a decomposition tree T of 
span less than 3- \G+\, whose root is labelled by (w, (w, w + )) 
for some extended limit- word (w, w_|_) G G+- 

According to the second part of Lemma (|9), and since there 
are less ./-classes than there are elements in the monoid Q + , 

the depth of T is at most 3 • |£+| 2 . (10) 

To complete the proof of Lemma [7] we prove that for every 
node t labelled (u, (u, u+)) of depth h in the decomposition 
tree and for all states s,t G Q, 

u+(s,t) = l > , (11) 

u(s,t) = 1 => u{s,t) >pi h in . (12) 

We prove dTTT > and ([T2l by induction on h. 

If h = then the node is a leaf, hence u is a letter a and 
u = u + = a. Then (fTTT > holds by definition of a and ([T2l i 
holds by definition of p m \ n . 

For the induction, there are two cases. 

First case, t is a concatenation node labelled by 
(it, (u, u+)) with two sons labelled by (iti, (ui, u+.i)) and 
(112, (u2,u + 2)). We first prove that ([TTt holds. Let s,t G Q 
such that u+(s,£) = 1. By definition of a decomposition 
tree, u + = u + .i • u + 2- Since u+(s, t) = 1 then by 
definition of the concatenation there exists q G Q such that 
u+.i(s, q) — 1 and u+^g, i) = 1. By induction hypothesis 
we have u±{s, q) ■ U2(q,t) > 0; and since it = u% ■ u% 
then u(s,t) > ui(s,q) ■ U2(q,t), which proves the direct 
implication of (fTTT >. The converse implication is similar: if 
u(s,t) > then by definition of matrix product, there exists 
q G Q such that u\(s,q) > and u(q,t) > 0, and 
we use the induction hypothesis to get u+(s, t) = 1. This 
concludes the proof of dTTT >. Now we prove that ([T2l holds. 
Let s, t G Q such that u(s, t) = 1. By definition of a 
decomposition tree, u = Ui • 112. Since u(s,t) = 1 then 
by definition of the product of two limit-words there exists 
q G Q such that Ui(s,q) = 1 and u.2(q,t) = 1. Then 

u(s,t) > ui(s,q) ■ u 2 (q,t) > #U ' Pt^ = Pmin where the 
first inequality is by definition of the matrix product and the 
second inequality is by induction hypothesis. This completes 
the proof of <fl2l l. 

Second case, t is an iteration node labelled 
by (it, (u", u+)) with k sons ti,...,tk labelled by 
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(u, u+)), . . . , (uk, (u, u + )). The proof that ( fTTT i holds 
is similar to the concatenation node case (and relies on the 
fact that u + is idempotent). We focus on the proof of ( [T2| >. 
Let s,r € Q such that u'(s, r) = 1. By definition of a 
decomposition tree, u = ui • • • Since t is an iteration 
node, k > 3 thus: 

u(s, r) > mi(s, r) ■ ^2 (("2 • • • u k -i) (r, q) ■ u k (q, r)) . 

(13) 

To establish (PL?! we prove that: 

ui(s,r) > p^" in , (14) 
Vg 6 Q, (w2---Wfc-i)(r,g) > => u*;(g, r) > p, 2 nin . (15) 

First we prove (fT4l . Since u"(s, r) = 1 then by definition 
of the iteration operation, r is u-recurrent and u(s, r) = 1. 
By induction hypothesis applied to t\, according to (fl2] |. it 
implies Ui(s, r) > p^ in i.e (fl4l . 

Now we prove (Q3). For that we use the hypothesis 
that (u, u+) is not a leak witness. Let q G Q such that 
(u 2 ■ ■ ■ Mfc_i)(r, q) > 0. Then by induction hypothesis applied 
to t%, . . . according to (JTTJ, u^T 2 (r,q) = 1. Thus 

by idempotency of u + , u + (r, g) = 1. Since by hypothesis 
u"(s, r) = 1 then r is u-recurrent and since (u, u+) is not a 
leak witness then necessarily u(q, r) = 1. Thus, by induction 
hypothesis and according to ( TTSi i. Uf.(q,r) > p^ in i-e dl5t . 

Now, putting ( fT3] l. ( [Pil l and (fTBI l altogether, 

u(s,r) > ui(s,r) ■ ^2(u 2 ■ ••Ufe_i)(r,g) • u k (q,r) 
<?e<3 

„h+i 

where the second inequality holds because 
^ ge q(u 2 ■ • -Ufc_i)(r, q) = 1 by basic property of transition 
matrices. This completes the proof of (fT2l i. 

To conclude, according to (TlOb the depth of a decomposition 
tree can be bounded by 3 • |{?+| 2 , and since Q + has less 
than J = 2 2 I'2I elements the depth h is less than 3 • J 2 . 
Then according to (fTTT l and (fT~2t this completes the proof of 
Lemma [7] ■ 

V. A FEW LEAKTIGHT AUTOMATA 

In this section, we present several properties and examples 
of leaktight automata. 

A. Two basic examples 

The automaton on Fig. [2] is leaktight. Its extended Markov 
monoid is depicted on the right-hand side (except for the 
neutral element (1,1)). Each of the four directed graphs 
represents an extended limit-word (u, u + ); the edges marked 
+ are the edges that are in u+ but not in u. 

The initial state of the automaton is state 0, and the unique 
final state is state 1. This automaton has value 1 and this can 
be checked using its Markov monoid: there is a single value 1 



witness a". Notice that the two distinct extended limit- words 
labelled by a" and b • a' on Fig. [2] correspond to the same 
limit- word a'. 




Fig. 2. A leaktight automaton and its extended Markov monoid. 

The automaton on Fig. [3] is leaktight. The initial state of 
the automaton is state 0, and the unique final state is state F. 
The Markov monoid has too many elements to be represented 
here. This automaton does not have value 1. 




Fig. 3. A leaktight automaton which does not have value 1. 

B. The class of leaktight automata is rich and stable 

The class of leaktight automata contains all known classes 
of probabilistic automata with a decidable value 1 problem, 
in particular hierarchical automata defined in [3] and ((-acyclic 
automata defined in lfl3l . 

Proposition 4. Deterministic automata, hierarchical proba- 
bilistic automata and ^-acyclic automata are leaktight and 
these inclusions are strict. 

Another interest of the class of leaktight automata is its 
stability under two natural composition operators: parallel 
composition and synchronized product. An automaton „4||£> 
is the parallel composition of two automata A and B if its 
state space is the disjoint union of the state spaces of A and 
B plus a new initial state. For every input letter, the possible 
successors of the initial state are itself or one of the initial 
state of A and B. An automaton A x B is the synchronized 
product of two automata A and B if its state space is the 
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cartesian product of the state spaces of A and B, with induced 
transition probabilities. 

Proposition 5. The leaktight property is stable by parallel 
composition and synchronized product. 

C. About ^-height 

The jj-height of an automaton is the maximum over all 
elements u of its Markov monoid of the minimal number of 
nested applications of the iteration operator needed to obtain 
u. As already mentioned, an adaptation of a result by Kirsten 
(Lemma 5.7 in ifTBTl ) shows that the ft-height of an automaton 
is at most \Q\. A natural question is whether this bound is 
tight. The answer is positive, a simple computation shows 
that the only value 1 witness of the automaton of Fig. [4] 
is u = (• • • ((ajai)*a2) tt a 3 ) tl • • • a n -i)K whose tf-height is 
n=\Q\-2. 




(a;);>o 



Fig. 4. A leaktight automaton with value 1 and ((-height n. 

The following proposition shows a crucial difference be- 
tween leaktight automata and jj-acyclic automata. 

Proposition 6. Deterministic automata and ^-acyclic au- 
tomata have ^-height 1. 

Conclusion 

We introduced a subclass of probabilistic automata, called 
leaktight automata, for which we proved that the value 1 
problem is PSPACE-complete. 

In the present paper we considered automata over finite 
words. Next step is the adaptation of our results to infinite 
words and probabilistic Biichi automata (TJ, 0, as well as 
partially observable Markov decision processes. 

A natural question is "what does the Markov monoid say 
about a probabilistic automaton (leaktight or not)?". Since 
the Markov monoid is independent of the actual values of 
transition probabilities (only positivity matters), this suggests 
the two following questions. Given a probabilistic automaton 
whose transition probabilities are unspecified (only positivity 
is specified), 

1) is it decidable whether the answer to the value 1 problem 
is the same for any choice of transition probabilities? 



2) does the Markov monoid contain a value 1 witness if 
and only if the automaton has value 1 for some choice 
of transition probabilities? 
The first question, suggested by a referee of the present paper, 
is open, while the answer to the second question seems to be 
negative. 
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Appendix 
Tool lemmata 

We start with a few tool lemmata. 

Lemma [T] The following two properties hold: 

• For every word u G A*, the word is idempotent. 

• For every limit-word u G {0, 1}^ , the limit-word u'^' ! is idempotent. 

Proof: The second statement implies the first one, so we prove the second one. 

Let n = \Q\ and s,t G Q such that u n (s, t) = 1. We want to prove that u 2 ' n -(s,t) = 1. Since u n (s, t) = 1, there exists 
g £ Q and A;,Z < |Q| such that u fc (s, g) = 1, u k ~ l (q,q) = 1 and u (q, t) = 1. Consequently, there exists fc' < |Q| such 
that u fc (q, q) = 1, and since k'\n\, this implies u n (g,?) = 1, thus u 2 ' n -~ k ~ l (q, q) = 1 and finally u 2 '"(s,i) = 1. 

The proof that u 2 ' n -(s,t) = 1 implies u n '(s,t) = 1 is similar. ■ 

The following lemma provides two simple yet useful properties. 

Lemma 10. Let A be a probabilistic automaton. 

i) Let u be an idempotent limit-word. Then for each state s G Q, there exists t G Q such that u(s, t) = 1 and t is 
u-recurrent. 

ii) Let (v, v + ) be an extended limit-word of the extended Markov monoid of A. Then: 

Vs,f G Q, (v(s,i) = 1 => v+(s,t) = l) . (16) 

Proof: We prove i). Let C C Q be a strongly connected component of the graph u reachable from s. (Recall that u can 
be seen as the directed graph whose vertex set is Q and where there is an edge from s to t if u(s,t) = 1.) Then for every 
t,q G C there exists k and a path t = t\,t2, ■ ■ ■ ,tk — Q from t to q in u. Thus, (u fc )(£,<7) = 1 and since u is idempotent, 
u(t,q) = 1, Thus, C is a clique of the graph u and all states of C are recurrent. Since C is reachable from s in u, the same 
argument proves that for every t G C, u(s, t) = 1, 

We prove ii). By definition, the extended Markov monoid is the smallest monoid containing {(a, a) a G ^4} and (1, 1), 
stable by concatenation and iteration. Property (fT6l holds for every pair (a, a) where a £ A. Moreover this property is stable 
by concatenation and iteration. This completes the proof. ■ 

The value 1 problem depends quantitatively on the transition probabilities 
In this section, we give a short proof of the following lemma, claimed about Fig [T] 
Lemma 11. The probabilistic automaton A has value 1 if x > |. 
We first note: 

Pa{0 ^ T ) = l - ■ x n and P^(0 ^ _L) = \ ■ (1 - x) n 

Fix an integer N and consider the following stochastic process: it consists in (at most) N identical rounds. In a round, there 
are three outcomes: winning with probability p n = | • x n , losing with probability q n = | • (1 — x) n , or a draw with probability 
1 ~ (Pn +9n)' Once the game is won or lost, it stops, otherwise it goes on to the next step, until the N th round. This stochastic 
process mimics the probabilistic automaton reading the input word (ba n ) N . 

The probability to win is: 

P(Winjv) = J2k=i P(Win at round fc) 

_ „ l-(l-(p„+q„)) JV 

= ^-(l-a-CPn + <?„))") 

- i . ft n ( m , „ 



(i-(i-( P „ + (Z „)r) 



We now set N = n. A simple calculation shows that the sequence (1 — (1 — (p n + q n )) n )ne^ converges to 1 as n goes 
to infinity. Furthermore, if x > | then < 1, so ^ = ("^j 2 ) converges to as n goes to infinity. It follows that the 
probability to win converges to 1 as n goes to infinity. Consequently: 

limP^(0 T) = 1 . 
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Undecidability in a very restricted case 



In this section we sharpen the undecidability result of the value 1 problem to probabilistic automata having only one 
probabilistic transition. 

We say that a probabilistic automaton is simple if for all letters a, for all states s and t, we have AI a (s,t) G {0,^,1}. 
According to |fl3l , the value 1 problem is undecidable for simple probabilistic automata. We first show how to simulate a 
(simple) probabilistic automaton with one having only one probabilistic transition, up to a regular language: 

Proposition 7. Given a simple probabilistic automaton A = (Q, A, (M a ) agj 4, qo, F), there exists a simple probabilistic 
automaton B over a new alphabet B, with one probabilistic transition, and a morphism 2 : A* — > B* such that: 

Vw g A*,P A (w) = V B (w). 

The morphism 2 will not be onto, so this simulation works up to the regular language A* = {w \ w G A*}. We shall 
see that the automaton B will not be able to check that a word read belongs to this language, which makes this restriction 
unavoidable in this construction. 

We first give the intuitions behind the construction. Intuitively, while reading the word w, the probabilistic automaton A 
"throw parallel threads". A computation of A over w can be viewed as a tree, where probabilistic transitions correspond to 
branching nodes. 

a b b a b b 




Fig. 5. An example of a computation. 



On the figure, reading a from qo or b from qi leads deterministically to the next state. Reading b from q2 leads at random to 
r or to s, hence the corresponding node is branching. Our interpretation is that two parallel threads are thrown. Let us make 
two observations: 

> threads are not synchronized: reading the fourth letter (an a), the first thread leads deterministically to the next state, 
while the second thread randomizes; 

> threads are merged so there are at most n — \Q\ parallel threads: whenever two threads synchronize to the same state q, 
they are merged. This happens in the figure after reading the fifth letter (b). 

The automaton B we construct will simulate the n threads from the beginning, and take care of the merging process each 
step. 

Proof: We denote by the states of A, i.e Q = {qo, . . . , g„_ i}. The alphabet B is made of two new letters '*' and 
'merge' plus, for each letter a G A and state q € Q, two new letters check(a, q) and apply(a, q), so that: 

B = {*, merge} U [J {check(a, q), apply(a, q)} 

a£A,q£Q 

We now define the automaton B. We duplicate each state q G Q, and denote the fresh copy by q. Intuitively, q is a temporary 
state that will be merged at the next merging process. States in B are either a state from Q or its copy, or one of the three 
fresh states s*, so and si. 

The initial state remains qo as well as the set of final states remains F. 

The transitions of B are as follows: 

• for every letter a £ A and state q G Q, the new letter check(a, q) from state q leads deterministically to state i.e 

-^check(a,q) (?) = s *' 

« the new letter * from state s* leads with probability half to so an d na lf t° si, i-e Af s> (*) = |so + (this is the only 
probabilistic transition of B); 

m the new letter apply(a,q) from states so and si applies the transition function from q reading a: if the transition M a (q) 
is deterministic, i.e M a (q,r) = 1 for some state r then A^ a ppiy(a.?) (so) = t and Af app i y ( ai9 )(si) = f, else the transition 
M a (q) is probabilistic i.e M a (q) — \r + \r' for some states r,r\ then M apply ( a g ) (s ) = f and M apply ( a 9 )(si) = r'\ 

• the new letter merge activates the merging process: it consists in replacing q by q for all qeQ. 
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apply(a,g) ' r ° merge 



k apply(a, q) merge 
si ► ri ► ri 

The first gadget. 



Whenever a couple (letter, state) does not fall in the previous cases, it has no effect. The gadget simulating a transition is 
illustrated in the figure. 

Now we define the morphism 2 : A* — > B* by its action on letters: 

2 = check(a, q ) ■ * ■ apply(a, q ) . . . check(a, q n -i) ■ * • apply(a, q n -i) • merge. 

The computation of A while reading w in A* is simulated by B on w, i.e we have: 

F A (w) = F B (w) 

This completes the proof. ■ 
Let us remark that B is indeed unable to check that a letter check(a, q) is actually followed by the corresponding apply(a, q): 

in-between, it will go through s* and "forget" the state it was in. 

We now improve the above construction: we get rid of the regular external condition. To this end, we will use probabilistic 

automata whose transitions have probabilities 0, |, § or 1. This is no restriction, as stated in the following lemma: 

Lemma 12. For any simple probabilistic automaton A = (Q, A, (M a ) a ^A,qo, F), there exists a probabilistic automaton B 
whose transitions have probabilities 0, |, § or 1, such that for all w in A*, we have: 

val(A) = val(B). 

Proof: We provide a construction to pick with probability half, using transitions with probability 0, |, | and 1. The 
construction is illustrated in the figure. 



2 




Fig. 7. Simulating a half with a third. 

In this gadget, the only letter read is a fresh new letter jj. The idea is the following: to pick with probability half tq 
or r\, we sequentially pick with probability a third or two thirds. Whenever the two picks are different, if the first was a 
third, then choose tq, else choose r\. This happens with probability half each. We easily see that P^(ao • ai • . . . dfc-i) = 
su Pp P e (a -f -ai -f ...Ofc-i T )■ 

■ 

Proposition 8. For any simple probabilistic automaton A = (Q, A, (M a ) a£ A, qo, F), there exists a simple probabilistic 
automaton B over a new alphabet B, with one probabilistic transition, such that: 

val(A) = 1 val(B) = 1 . 

Thanks to the lemma, we assume that in A, transitions have probabilities 0, |, | or 1. The new gadget used to simulate a 
transition is illustrated in the figure. 

The automaton B reads words of the form u\ ■ finish • ■ finish . . ., where 'finish' is a fresh new letter. The idea is to "skip", 
or "delay" part of the computation of A: each time the automaton B reads a word U{, it will be skipped with some probability. 

Simulating a transition works as follows: whenever in state s*, reading the letter leads with probability one third to sx, 
one third to sq and one third to w. As before, from sq and sx, we proceed with the simulation. However, in the last case, we 



check(a, q) 



Fig. 6. 
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check(a, q) 




to g 



apply(a,g) _ 

so to 



1 apply(a,g) n 



finish 



Fig. 8. The second gadget. 



"wait" for the next letter 'finish' that will restart from go. Thus each time a transition is simulated, the word being read is 
skipped with probability i. 

Delaying part of the computation allows to multiply the number of threads. We will use the accepted threads to check the 
extra regular condition we had before. To this end, as soon as a simulated thread is accepted in £>, it will go through an 
automaton (denoted C in the construction) that checks the extra regular condition. 

Proof: We keep the same notations. The alphabet B is made of three new letters: 'merge' and 'finish' plus, for each 
letter a £ A and state q £ Q, two new letters check(a,g) and apply(a, q), so that: 

B = {*, merge, finish} U {check(a, q), apply(a, q)} 

a£A,qGQ 

We first define a syntactic automaton C. We define a morphism 2 : A* — > B* by its action on letters: 
a = check(a, go) ■ * • a Pply(a, Qo) ■ ■ ■ check(a, q n -x) • * • apply(a, q n -i) ■ merge. 

Consider the regular language L = {w ■ finish | w £ A*}*, and C — (Qc,Sq, sc,Fc) an automaton recognizing it. 

We now define the automaton B. We duplicate each state q € Q, and denote the fresh copy by q. States in B are either a 
state from Q or its copy, a state from Qc or one of the four fresh states s*, s , s\ and wait. 

The initial state remains qo, and the set of final states is Fq. 

The transitions of B are as follows: 

• for every letter a £ A and state q £ Q, the new letter check(a, q) from state q leads deterministic ally to state s* i.e 

-Wcheck(a.g) (?) = s *' 

> the new letter * from state s* leads with probability one third to s*, one third to s an d one third to s\, i.e M s , (*) = 
|s* + ^so + |si (this is the only probabilistic transition of £>); 

• any other letter from state s* leads deterministically to w, i.e M St (_) = wait; 

• the new letter apply (a, q) from states sq and si applies the transition function from q reading a: if the transition M a (q) 
is deterministic, i.e M a (q,r) — 1 for some state r then Af app i y ( a 9 )(so) = f and M app i y ( aj9 )(si) = f, else the transition 
M a (q) is probabilistic i.e M a (q) = \r + \r' for some states r,r', then M apply ( o g) (s ) = f and M apply ( Q ? )(si) = r'; 

• the new letter merge activates the merging process: it consists in replacing q by q for all q £ Q; 

• the new letter finish from state wait leads deterministically to go; 

• the new letter finish from state g in F leads deterministically to sc\ 

• the new letter finish from any other state is not defined (there is a deterministic transition to a bottom non-accepting state). 
Transitions in C are not modified. Whenever a couple (letter, state) does not fall in the previous cases, it has no effect. 

We now show that this construction is correct. 

We first prove that for all w £ A*, there exists a sequence of words (u> p ) p >i such that F^(w) = sup p Pg(u> p ). 
The probability to faithfully simulate one transition is (§)"■ It follows: 

6 B (q ,w ■ finish) = (0 F A (w)+U- 

for some k satisfying n < k < n ■ \w\ (the number k corresponds to the number of transitions followed along a faithful 
simulation of w). This implies that sup p Pg((w • finish) p ) = F^(w), hence if val(^4) = 1, then val(£>) = 1. 

Conversely, we prove that if val(B) = 1, then val(A) — 1. Let w a word read by B accepted with probability close to 1, we 
factorize it as follows: w = U\ ■ finish • . . . • Uk ■ finish, such that «, does not contain the letter finish. The key observation is 
that if k = 1, the word w is accepted with probability at most |. Hence we consider only the case k > 1. We assume without 
loss of generality that P^t(ui) > (otherwise we delete u\ ■ finish and proceed). In this case, a thread has been thrown while 
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reading u\ that reached sq, so the syntactic process started: it follows that u, for i > 1 are in the image of 2- This implies 
that the simulation is sound: from w we can recover a word in A* accepted with probability arbitrarily close to 1 by A. 
This completes the proof. ■ 
The proposition implies the following corollary: the value 1 problem is undecidable, even for probabilistic automata with 
only one probabilistic transition. 

Decomposition trees of bounded span 

Theorem |7] Let A be a finite alphabet, [M, ■, 1) a monoid equipped with a function (J that maps every idempotent e £ M to 
another idempotent element e" £ M and (f> : A* — » M a morphism. Every word u £ A + has a decomposition tree whose span 
is less than 3 • |M|. 

The following proof appeared in ||221 : we give it here for the sake of completeness. 

Proof: We start by adding a few letters to A. For every idempotent e £ M, we add a letter e to the alphabet A, and extend 
4> by 0(e) = e. We do not lose generality because this operation does not modify the monoid M, and a decomposition tree for a 
word with letters from the original alphabet cannot use the new letters of the extended alphabet A = A\J {e | e £ M, e = e 2 }. 

We proceed by induction on the length of u. 

First, if u is a letter a, the decomposition tree whose only node is labelled by (a, <fr{a)) has span 1. 

Consider now a word u with at least two letters. According to Theorem |6] there exists a Ramseyan factorization tree T u of 
u of height less than 3 • |M|. According to Lemma [8] this factorization tree is in general not a decomposition tree, except in the 
very special case where it has no discontinuous nodes. The rest of the proof shows how to transform T u into a decomposition 
tree of span less than 3 • \M\. The proof technique consists in taking care of the discontinuous nodes of T u in a bottom-up 
fashion. 

If T u has no discontinuous node then it is already a decomposition tree. Moreover its span is equal to its height, which is 
less than 3 • \M\. 

If T u has at least one discontinuous node, let t be such a node with maximal depth. By definition, t has at least three 
children t\, . . . , tk labelled by (vx, (f>(vi)), . . . , (vk, 4>{vk)) an d t itself is labelled by (v, <j>(v)) = (vx ■ ■ ■ Vk, <t>( v i ■ ' • v k))- We 
have 4>(vi) = . . . = (j>(Vk) = e and since t is discontinuous, e ^ e\ We distinguish two cases. 

• Either t is the root of T u . In that case v = u and we can construct directly a decomposition tree T u of u of span less 
than 3 • |M|. Let T%, . . . , Tk be the subtrees of T u whose roots are respectively t%, . . . ,tk. Then, since t is a discontinuous 
node of maximal depth in T u , each subtree T%,...,Tk contains no discontinuous node at all. Consequently, each subtree 
Ti is a decomposition tree whose span is equal to its height, which is less than 3 ■ \M\ — 1. Then a decomposition tree 
for it is the tree with the root labelled by (u, er) and children Ti, . . . , Tk- Since e ^ er, the root is discontinuous and the 
span of this tree is less than 3 • \M\. 

• Or t is not the root of T u . Since t is labelled by (v, e"), there exist two words w, w' £ A + such that u = w ■ v ■ w'. We 
replace the subword v in u by the letter (£ and obtain the word v! = w ■ • w' . Since t is a discontinuous node, it is 
an iteration node and has at least three children, thus v has length at least 3. Thus, u' is strictly shorter than u and we 
can apply the induction hypothesis to vf: let T 1 be a decomposition tree for u', whose span is less than 3 • \M\. One of 
the leaves of T" corresponds to the letter (£ of u' and is labelled by (e|, e"). We replace this leaf by the decomposition 
tree T v of v given by induction hypothesis. Since the root of T v is labelled by (v, e"), we obtain a decomposition tree 
for u — w ■ v ■ it/, whose span is less than 3 • \M\. This completes the induction step. 

■ 

Lemma |9] Let A be a finite alphabet, and M a monoid equipped with a function (j that maps every idempotent e £ M to 
another idempotent element & £ M. Suppose moreover that for every idempotent e £ M, 

e" • e = e" = e • e" . (17) 

Then for every idempotent element e £ M, either e* = e or e' <j e. 

As a consequence, the number of discontinuous nodes along a path in a decomposition tree is at most J, where J is the 
number of J' -classes of the monoid. 

Proof: We prove the first part of the lemma. Equation (|9) implies that e" = e"ee" thus e" <j e. Now, we suppose that 
e <j e' and prove that e = e". Since M is finite, we have eVeK Since e • e" = e", it follows elZeK By a dual argument, 
we have e£e"; hence eT-LeK Both e and e" and idempotents, so according to Green's theorem (see e.g lfT4ll . Theorem 2.2.5.) 

e = ek 

The second part of the lemma is an immediate consequence, since the sequence of elements of the monoid labelling a branch 
of a decomposition tree, starting from the root, is non-decreasing for the J^-order and strictly increasing on discontinuous nodes. 
Indeed, if an internal node has two children labelled by (ui,ui) and (^2,112) then by definition of a decomposition tree the 



16 



node is labelled by (ui ■ U2,ui ■ U2) and by definition of the J"-order, Ui ■ U2 <j Ui and Ui ■ U2 <j U2. For continuous 
nodes, the same idempotent labels both the internal node and its children. For discontinuous nodes, the father node is labelled 
by some idempotent e" and the children by e ^ e", and we conclude with supra. ■ 

About the extended Markov monoid and leak witnesses 
Theorem |5] An automaton A is leaktight if and only if its extended Markov monoid contains no leak witness. 

The proof is split in two parts, the direct implication (Lemma [T31 and the converse implication (Lemma [T4li. 
Lemma 13. If the extended Markov monoid of an automaton A contains a leak witness then A has a leak. 

Proof: Suppose that there is a leak witness (u, u + ) in the extended Markov monoid. 

By definition of a leak witness, u and u + are idempotent and there exists r,q £ Q such that r is u-recurrent, u + (r, q) = 1 
and u(q, r) — 0. We prove now that there exists a leak from r to q. 

By induction, following the proof of Lemma [4] for each (u, u+) in the extended Markov monoid, we build a sequence 
(u n ) n< zfa such that for every states s,t £ Q the sequence (it„(s, i))neN converges and 

u(s,i) = l lim u n (s, t) > , (18) 

n 

u+(s,t) = <^=> Vn £ N,u n (s,t) = , (19) 
u+(s,t) = 1 Vn £ N,u n (s,t) > . (20) 

To complete the proof, we show that (u„) 71 gN is a leak in A from r to q. According to Definition [5] there are four conditions 
to be met. 

First condition is by hypothesis. Moreover, since u + is idempotent then according to ( fT8l all the (u n ) ne ^ are idempotent. 
Second, let M. u the Markov chain associated with transition probabilities {u{s, t)) s ( G q. We prove that r is M. u -recurrent. 
Since r is u-recurrent, and according to ([TBI , Vs £ Q,u(r,s) > u(s,r) > 0. But u is idempotent because u is 

idempotent and ( fT8l ). Thus according to Lemma [2] r is u-recurrent. Third, we need to prove Vn £ N,u n (r,q) > 0, this holds 
because of d20T ) and u+(r, q) = 1, Fourth, we need to prove that r is not reachable from q in M u . Since u = lim„ u n and 
according to ( fT8l , u(s,t) = 1 <S=^> u(s,t) > 0, thus accessibility in the directed graph u and in the Markov chain M. u 
coincide. Since u(r, q) = and u is idempotent, r is not accessible from q in u, thus neither accessible in Ai u . ■ 

Lemma 14. /f f/ie extended Markov monoid of an automaton A contains no leak witness then A is leaktight. 

Proof: By contraposition, suppose there is a leak (u n ) ne m from a state r to a state g in A, and for each s,t £ q, denote 
u(s,t) the limit of the sequence (u n (s,i)) n ^ and A4 U the Markov chain induced by (it(s, t)) s> t^Q- By definition of a leak: 

r is recurrent in A^ u , (21) 
Vn £ N,u«(r,9) > 0, (22) 
r is not accessible from g in A^^. (23) 

To get to the conclusion, we use the leak (u„) ii6 n to build a leak witness (v,v + ) in the extended monoid Q + of A. 
The first task is to define the pair (v, v+). By hypothesis, we can apply the Lemma [7] to each word u n of the leak, which 
gives for each n £ N a pair (u n , u+.„) € Q+ such that for all states s, t: 

u + , n (s,t) = 1 <=^ u n (s,t) > , (24) 

u„(s,i) = l =S> u n (s,t) > p, 2 nin . (25) 

Since the extended Markov monoid is finite, there exists N £ N such that: 

for infinitely many n e N, (ujv, u +,jv) = (u„ ,u^„) . (26) 

Let (v,v + ) = (u^r, u + jv)' e+ ' ! . Then according to Lemma [T] (v, v+) is idempotent. Note also that according to (124-b and 
since the u n are idempotent (by definition of leaks), 

u + jv is idempotent and v + = u + jv . (27) 

Now, we prove that (v, v+) is a leak witness. According to i) of Lemma [Tol since v is idempotent, there exists r' such that 
v(r,r') = 1 and r' is v-recurrent. By definition of a leak witness, if we prove that (a) v + (r',q) = 1, (b) v(q,r') = then 
(v,v + ) is a leak witness. 
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We first prove (a). Let rj = p^ in and K = \G+\\. Then: 

v(r, r ) = 1 
=► uf(r,r') = l 

=>■ u„ (r, r') = 1, for infinitely many n 
u^(r,r r ) > 77 , for infinitely many n 

=> M K (r,r') > ?7 K 
r' is u-recurrent 

=> 3l,u l (r',r) > 
(because r and r' are in the same class of u-recurrence) 

=>> 3l,u l+1 (r',q) > 

=>> 3l,3N , yn>N',u l + 1 (r',q)>0 

=> 37V', Vn > N',u n (r',q) > 
=>■ 3JV',Vn>W / ,u+, n (r',<j) = l 
=► u +:A r(r',g) = 1 

v + (r',q) = 1 

Now we prove (b). By contradiction, suppose that v(q,r') = 1. Then: 
v(q,r') = 1 

=>■ u^(g, r') = 1, for infinitely many n 
=> u„ (g, r') > 7/ , for infinitely many n 

=>■ r' is reachable from g in A^u 
==>■ ?* is reachable from q in Ai u 



(by definition of r') 
(by definition of v) 
(by definition of N) 
(by 423) 

(because u = limu„) 

n 

(because r is u-recurrent) 



(because u(r. q) > 0) 
(because u = lim u„) 

n 

(the u n are idempotent) 
(by (El) 
(by definition of AT) 
(according to (l27l>). 



(by definition of v) 
(by definition of N) 
(by 4Z3) 
(because u = lim u n ) 

n 

(according to d28l l) 



(28) 



which contradicts d23l >. 

This completes the proof of (b), thus (v, v+) is a leak witness, which concludes the proof of Lemma [141 ■ 

About ((-height 

In this section, we adapt Kirsten's results to show that the (J-height of an automaton is at most \Q\. The following statements 
are straightforward translations from [15 , we provide them for the sake of completeness. 

Consider u an idempotent limit-word. We define ^ u the relation on Q by i ~ j if u(i,j) = 1 and u(j, i) = 1. Clearly, ^ u 
is symmetric, and since u is idempotent, ^ u is transitive. If for some i there is a j such that i ~ u j, then i ~ u i thanks to 
u's idempotency. Consequently, the restriction of ^ u to the set 

Z u = {i S Q I there is some j such that i ^ u j} 

is reflexive, i.e ^ u is an equivalence relation on Z u . By equivalence class of ^ u we mean an equivalence class of ~ u on Z u . 
We denote by [i] u the equivalence class of i, and by Cl(u) the set of equivalence classes of ~. 

Lemma 15. The following two properties hold: 

• Let u, v be two limit-words and i,j in Q. Then (u • v)(i, j) > u(i, k) ■ v(fc, j) for all k in Q. 

• Let u be an idempotent limit-word and i,j in Q. There is some I in Q such that u(i,j) = u(«, I) ■ u(l, I) ■ u(l,j). 

Proof: The first claim is clear and follows from the equality: 

(u-v)(i,j) = J2 u (h k ) -v(k,j) . 

keQ 

Consider now the second claim. For every k, we have u(i,j) — (u 3 )(i,j) = X^zi'eQ u (*'0 ' U (M') ' u (l\j) > u(i,k) ■ 
u(k,k) ■ u(k,j). Since u = u n+2 , there are i — io, . . . ,i n +2 — j such that u(i,j) = u(io,ii) • . . . • u(i n +i, i n +2)- By a 
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counting argument, there are 1 < p < q < (n + 1) such that i p = i q . Let I = i p . We have 

u(i, I) = u p (i, I) > u(i , ii) ■ u(i p -i,i p ), 
u(l,l) = u q - p (l,l) > u(i p ,i p+ i) . ..u(i q -i,iq), and 
u(i.j) = u n+2 - q (l 7 j) > u(i q ,i n+2 ) . . .u(i n+1 ,i n+2 ) . 

Hence, u(i,l) ■ u(l,l) ■ u(Z, j) > u(io,ii) . . .u(i p -i,i p ) = u(i,j), and the second claim follows. ■ 

Lemma |6] Let u and v be two idempotent limit-words. Assume u <j v, then |Cl(u)| < |Cl(v)|. 

Proof: Let a, b two limit-words such that a • v • b = u. We assume a • v = a and v • b = b. If a and b do not 
satisfy these conditions, then we proceed the proof for a = a • v and b = v • b. We construct a partial surjective mapping 
j3 : Cl(v) — > Cl(u). The mapping f3 depends on the choice of a and b. For every i,j with i ~ v i and j ^ u j satisfying 
a(j, i) ■ v(i, i) ■ h(i,j) = 1, we set f3([i] v ) = [j] v . To complete the proof, we have to show that j3 is well defined and that /3 
is indeed surjective. 

We show that f3 is well defined. Let i, i' such that i ^ v i and i' ^ v i'. Moreover, let j,j' such that j ^ u j and j' ^ u j'. 
Assume a(j, i) ■ v(i,i) • b(i,j) — 1 and ■ v(i',i') ■ h(i',j') = 1. Thus, f3([i} v ) = [j] u and (3([i'] v ) = [j'] u . To show 

that /3 is well defined, we have to show that if [i] v — [i'] v , then [j] u — [j'] u . Assume [i] v = [i'] v , i.e, i ^ v i'. Hence, 
v(i,i') = 1. Above, we assumed a(j,i) ■ v(i,i) ■ b(i,j) = 1, and thus, a(j,i) = 1. Similarly, h(i',j') = 1. Consequently, 
a(j, i) ■ v(i, i') ■ h(i',j') — 1, i.e, u(j,f) = (a • v • b)(j, j') = 1. By symmetry, we achieve u(j' ,j) = 1, and hence, j ~ u j'. 

We show that j3 is surjective. Let j such that j ^ u j. We have to exhibit some i such that /3([i] v ) = b]u- Since j ~ u j, we 
have u(j,j) = 1. Since u = a • v • b, there are k, I such that a(j, k) ■ v(fc, I) ■ h(l,j) = 1, and in particular, v(fc, I) = 1. By 
Lemma [T5l there is some i such that v(k,i) ■ v(i,i) • v(i,l) — v(k,l) = 1, and in particular, v(i,i) = 1. We have a(j,i) = 
(a- v)CM) > a(j,k) ■ v(k,i) = 1, and b(i,j) = (v • b)(i,j) > v(i,l) ■ b(l,j) = 1. To sum up, a(j,i) ■ v(i,i) ■ b(i,j) = 1, 
and hence, /3([i] v ) = [j]u- ■ 

Lemma |5] Let u fee an idempotent limit-word. We have Cl(u") C Cl(u). Furthermore, if ^ u f/ien Cl(u") C Cl(u). 

Proof: Let i be such that i ~ u j i. We show [i] u = [i] u j. For every j with i ~ u j j, we have, i ^ u j. Hence, [i] u « C [i] u . 
Conversely, let j E [i] u ; we have u(i, j) = 1. Since i i, we have v?(i,i) = 1. To sum up, u"(i, j) = (u" • u)(i,j) > 
u"(i,i) ■ u(i,j) = 1, and by symmetry, v?(j,i) = 1, /.e, i ~ u j j. Hence j S [i] u «. 

Assume now u" 7^ u. Let i,j such that u(i,j) = 1 and u"(i,j) = 0. By Lemma [T51 there is some / such that u(i,j) = 
■ u(l,l) ■ u(l,j), so u(l,l) = 1. By contradiction, assume vr(l,l) — 1. Hence, 

u»(i,j) = (u • u» • u)(i,i) > u(i, I) • u«(l, /) • u(Z, j) = u(i, I) • u(Z, I) • u(/,j) = 1, 

j.e, u"(i, j) = 1 which is a contradiction. Consequently, u"(Z, Z) = 0, so Z ~ u Z and Z t^ u j Z. Thus, Z G Z u but Z ^ Z u ». Hence, 
there is a class [Z]„ in Cl(u), but there is no class [Z] u « in Cl(u"). In combination with the first part of this lemma, we obtain 
Cl(u») C Cl(u). ■ 

A FEW LEAKTIGHT AUTOMATA 
Proposition |4] Deterministic automata, hierarchical probabilistic automata and ^-acyclic automata are leaktight. 

Proof: It is obvious that deterministic automata are leaktight. We give an algebraic proof. For deterministic automata the 
iteration operation has no effect on limit-words. As a consequence, the extended Markov monoid only contains pair (u, u) 
whose both components are equal, and none of them can be a leak witness. The characterization given by Theorem [2] allows 
us to conclude that deterministic automata are leaktight. 

The proof for hierarchical automata is given in Proposition [9] 

The proof for jj-acyclic automata is given in Proposition [10] ■ 

Proposition [5] The leaktight property is stable by parallel composition and synchronized product. 

Proof: Both cases are proved easily. 
For the parallel product, let i be the new initial state. If there is a leak in A\\B from a state q ^ i then this a leak either 
in A or B. There can be no leak (u„)„ e N from i because i is w-recurrent only for those words u that are written with letters 
stabilizing i. 

For the synchronized product, the extended Markov monoid of the synchronized product A x B is the product of the extended 
Markov monoids of A and B. If there was a leak in A x B, then according to Theorem [5] there would be a leak witness 
(u, u + ) = ((u^t, ub), (u-)-^, u+ b)) in the extended Markov monoid of A x B from a state (r^rg) to a state (qA,qs)- 
Then r_4 is u^-recurrent and vl+,a{ta,<Ia) = 1 tnus since A is leaktight vla{1Aj r A) = !• Similarly, 113(53, T"b) = 1 thus 
u ((qa, Qb)-, i^Ai r B)) = 1 hence a contradiction. ■ 
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A. Leaktight automata strictly contain hierarchical automata 
The class of hierarchical automata has been defined in 0. 

The states Q of a hierarchical automaton are sorted according to levels such that for each letter, at most one successor is at the 
same level and all others are at higher levels. Formally, there is a mapping rank : Q — > [1, . . . , I] such that Va £ A, Vs, t £ Q 
such that a(s,t) > 0, rank(i) > rank(s) and the set {t | a(s,t) > 0,rank(t) = rank(s)} is either empty or a singleton. 

Proposition 9. Every hierarchical automata is leaktight. 

Proof: We prove by induction that for every extended limit- word (u, u+) in the extended Markov monoid of a hierarchical 
automata, for every state r: 

(r is u-recurrent) (Vq ^ r, u + (r, q) = 0) . (29) 

Property (l29t obviously holds for base elements (a, a). 

Property (l29l is stable by product: let (u, u+) and (v, v+) with property (l29~t and let r £ Q be uv-recurrent. By definition of 
hierarchical automata the recurrence classes of the limit- words u, v and uv are singletons thus r is necessarily both u-recurrent 
and v-recurrent. According to (|29~t , Vg ^ r, u + (r,t) = thus Vg ^ r, (u + v + )(r, q) = 0. 

Property ((29) is obviously stable by iteration, which terminates the proof. ■ 

The inclusion is strict, an example is given by Fig. [2] 

B. Leaktight automata strictly contain ^-acyclic automata 
The class of jj-acyclic automata has been defined in fl3l . 

Let A be a probabilistic automaton, to define (J-acyclic automata, we define an action on non-empty subsets of states. Given 
S C 2® and a letter a, by definition S ■ a = {t \ 3s £ S,a(s,t) = 1}. If S • a = S, then we define the iteration of a: 
S ■ a" = {t | Els £ S, a"(s, t) = 1}. Consider now the graph whose vertices are non-empty subsets of states and there is an 
edge from S , toTifS f -a = Tor5a = 5 and S ■ afi = T. The automaton A is jj-acyclic if the unique cycles in this graph 
are self loops. 

We extend the action on any limit-word: given S C Q and a limit-word u, by definition S ■ u = {t | 3s £ S, u(s, t) = 1}. 

Proposition |6] Deterministic automata, ^-acyclic automata and hierarchical automata have ^-height 1. 

Proof: For deterministic automata, this is obvious because there are no unstable idempotent in the Markov monoid so the 
iteration operation is useless and the jj-height is actually 0. 

For jj-acyclic automata, this is a corollary of results in iflJl : if a fj-acyclic automaton has value 1 then there exists a sequence 
of letters ao, bo, ai, . . . , a„, b n , a n +i £ (A U {e})* such that a feQai6* . . . anti^dn+i is a value 1 witness. ■ 

Proposition 10. Every ^-acyclic automata is leaktight. 

Proof: We prove that for all extended limit-word (u, u+), we have (u(s, t) = 0,u+(s,t) = 1) =>- s is transient, which 
implies the leaktight assumption, by induction on u. The case u = a is clear. Consider the case u = v", and let s,t states 
such that (u(s,t) = 0, u+(s,£) = 1). Then either (v(s,t) = 0,v+(s, t) = 1) or v(s,t) = 1 and t is transient in v. In the first 
case, the induction hypothesis ensures that s is transient in v. In the second case, s would be transient in v. In both cases, s 
is transient in v, so also in u. 

Consider now the case u = Ui • U2, and let s,t states such that (u(s,t) = 0,u + (s,t) = 1). Assume toward contradiction 
that s is recurrent in u. Let C = {q | u(s, q) = 1} be the recurrence class of s, so we have C ■ u = C. The jj-acyclicity implies 
that C ■ ui = C and C ■ u 2 = C. 

There are two cases: either there exists p such that (ui(s,p) = 0, Ui + (s,p) = 1), or such that u.i(s,p) = 1 and (u2(s,p) = 
0,ii2 + (s,p) = 1). Consider the first case, and T = C • u\. We have T C C, so T ■ vfi = C, which defines a fl-cycle over C, 
contradiction. In the second case, let T = C ■ u|, we have T C C so T ■ u" = C, which defines a jj-cycle over C, contradiction. 
This completes the proof. ■ 

The inclusion is strict: Fig. |4] provides an example of leaktight automaton which is not jj-acyclic. 
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